Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better
نویسندگان
چکیده
Federated learning (FL) enables distribution of machine workloads from the cloud to resource-limited edge devices. Unfortunately, current deep networks remain not only too compute-heavy for inference and training on devices, but also large communicating updates over bandwidth-constrained networks. In this paper, we develop, implement, experimentally validate a novel FL framework termed Dynamic Sparse Training (FedDST) by which complex neural can be deployed trained with substantially improved efficiency in both on-device computation in-network communication. At core FedDST is dynamic process that extracts trains sparse sub-networks target full network. With scheme, "two birds are killed one stone:'' instead models, each client performs efficient its own networks, transmitted between devices cloud. Furthermore, our results reveal sparsity during more flexibly accommodates local heterogeneity agents than fixed, shared masks. Moreover, naturally introduces an "in-time self-ensembling effect'' into dynamics, improves performance even dense training. realistic challenging non i.i.d. setting, consistently outperforms competing algorithms experiments: instance, at any fixed upload data cap non-iid CIFAR-10, it gains impressive accuracy advantage 10% FedAvgM when given same cap; gap remains 3% 2 times cap, further demonstrating efficacy FedDST. Code available at: https://github.com/bibikar/feddst.
منابع مشابه
Better Health, Less Spending
Cerebrovascular disease is a leading cause of disability in the United States and results in $20.6 billion in annual direct medical spending, with ≈700 000 ischemic strokes yearly. Despite advances in stroke prevention and treatment, deficits persist in healthcare value (defined as improved population health and patient experience at lower per-capita cost). Initiatives including the American He...
متن کاملBetter Schools, Less Crime?
I estimate the e ect of attending a rst-choice middle or high school on young adult criminal activity, using data from public school choice lotteries in CharlotteMecklenburg school district (CMS). Seven years after random assignment, lottery winners have been arrested for fewer and less serious crimes, and have spent fewer days incarcerated. Lottery winners attended schools that were higher qua...
متن کاملLayperson training for cardiopulmonary resuscitation: when less is better.
PURPOSE OF REVIEW Basic cardiopulmonary resuscitation, including use of automated external defibrillators, unequivocally saves lives. However, even when motivated, those wishing to acquire training traditionally have faced a myriad of barriers including the typical time commitment (3-4 h) and the number of certified instructors and equipment caches required. RECENT FINDINGS The recent introdu...
متن کاملOxygen – less is better , more is better ? ?
Altitude training has been employed as a tool for enhancing performance by athletes since the 1964 Mexico Olympics. The cost and impracticality of going to altitude regularly has led to the development of altitude simulation at sea level by creating low oxygen (‘hypoxia’) environments. ‘Altitude’ or ‘hypoxic’ training facilities are now springing up all around the country. The evidence supporti...
متن کاملذخیره در منابع من
با ذخیره ی این منبع در منابع من، دسترسی به آن را برای استفاده های بعدی آسان تر کنید
ژورنال
عنوان ژورنال: Proceedings of the ... AAAI Conference on Artificial Intelligence
سال: 2022
ISSN: ['2159-5399', '2374-3468']
DOI: https://doi.org/10.1609/aaai.v36i6.20555